Here are some thoughts on the remaining questions:

micromamba doesn't have a direct analog to conda --no-capture-input, and I'm not sure how its stream handling / attachement args correspond (if at all) so for now I've removed --no-capture-output from the command that's passed into the container. But not sure what the implications of that are.

I think this might prevent us from piping the nightly build STDOUT into a log file. If there is no way to stop micromamba from intercepting STDOUT we’ll have to find another way to capture the logs. We might be able to do this using Cloud Logging.

Installing the AWS CLI and Google Cloud SDK is hacky. Is AWS really only available via curl? gcloud can be installed similarly with curl, or via apt-get or from conda-forge (for OS X + Intel based Linux). What's the right approach? Also, how were we previously installing the Google Cloud SDK? I don't know what I changed to cause it not to be available any more.

Looks like AWS is not interested in publishing awscli2 on PyPI. I'm assuming the same goes for conda.

It looks like our google python packages were installed via other dependencies (gcsfs, pandas-gcp). I think we installed the CLI tool separately: Sometimes it is helpful to download the logs and data outputs of nightly builds when debugging failures. To do this you’ll need to set up the Google Cloud software Development Kit (SDK).

Install the gcloud utilities on your computer. There are several ways to do this. We recommend using conda or its faster sibling mamba. If you’re not using conda environments, there are other ways to install the Google Cloud SDK explained in the link above.

When do we want to trigger the creation of a new lockfile, and how do we make sure it incorporates newer versions of our dependencies that are outside of the current range?

• Should we let the dependabot keep bumping upper bounds and automatically create a new lockfile and merge it into those dependabot PRs?
• Should we automatically update conda-lock.yml any time pyproject.toml is changed?
• Should we remove all of the upper bounds on our dependencies and let the lockfile take the newest possible versions >of everything, and assume it all works if the tests pass? Would this in effect replace the need to run dependabot on our python dependencies?

• Option 3: Does this mean we would have an action that updates the lock files on a schedule? Would the logic be, run make conda-lock.yml , if the lock files changed, run the tests, if the tests pass, merge into main. If there are no new packages and the lock files do not change, don’t run the tests and don’t open a PR.
• Options 1 and 3 seem similar to me. I like option 1 because it lets dependabot figure out when it’s appropriate to update packages instead of running make conda-lock.yml on a schedule and figuring out if packages changed.
• Don’t we need to retain the upper bounds on our packages for when we know there are newer versions of dependencies that aren’t compatible with our code or other dependencies? For example, dagster requires an older version of coloredlogs and pydantic.
• I wonder if it’s possible to create an action that runs make conda-lock.yml any time the dependencies in pyproject.toml changes and commits the new files directly to the PR that changed pyproject.toml. This way the lock files are always updated when dependabot or a contributor bumps a version in pyproject.toml.

Do we want to build and install a catalystcoop.pudl package in our CI tests and Docker image, rather than installing from the repository directly?

• If so, do we want to continue distributing that package in PyPI and conda-forge?
• If so, what allowable dependency versions do we want to communicate to users of the package?

What are the benefits of installing from catalystcoop.pudl package instead of from the the repository directly? Distributing the package in PyPI and condo-forge seems like it will create unnecessary overhead.

Why does pip think that the recordlinkage version is 0.0.0 when it's installed with mamba?

I'm not sure! Happy to help debug.

Should we separate our interactive development dependencies (like jupyterlab) from the headless dev requirements? If we're going to install all of the optional dependencies in all of the conda environments, does it even make sense to break them out as optional dependencies?

Great point! It doesn't make much sense anymore to have optional dependencies if we treat PUDL like an application with pinned dependencies that produces data outputs. Does it make sense to keep the optional dependencies around until our main users no longer depend on install PUDL as a package?

I think this might prevent us from piping the nightly build STDOUT into a log file. If there is no way to stop micromamba from intercepting STDOUT we’ll have to find another way to capture the logs. We might be able to do this using Cloud Logging.

There's the micromamba --attach flag, but the docs are a bit sparse and I wasn't sure how to use it. I think we might be able to do micromamba --attach "" and have it not capture any outputs?

$ micromamba run --help                                                                                                                                                                        
Run an executable in an environment
Usage: micromamba run [OPTIONS]

Options:
  -h,--help                   Print this help message and exit
  -a,--attach TEXT            Attach to stdin, stdout and/or stderr. -a "" for disabling stream redirection
  --cwd TEXT                  Current working directory for command to run in. Defaults to cwd
  -d,--detach                 Detach process from terminal
  --clean-env                 Start with a clean environment
  -e,--env TEXT ...           Add env vars with -e ENVVAR or -e ENVVAR=VALUE
  --label TEXT                Specifies the name of the process. If not set, a unique name will be generated derived from the executable name if possible.

Looks like AWS is note interested in publishing awscliv2 on PyPI. I'm assuming the same goes for conda.

There's a stale user-contributed package on conda-forge but no, it seems like curl is the only way to programmatically install the AWS CLI which seems dumb. But oh well. The conda-forge version of google-cloud-sdk seems to be well maintained though, and installing it through via conda will allow us to have a specific, known version that's stored in the lockfile, so that's what I went with for now. And also it means that it'll be installed for all of us in pudl-dev so we don't all have to figure out how to install it independently on various platforms.

Options 1 and 3 seem similar to me. I like option 1 because it lets dependabot figure out when it’s appropriate to update packages instead of running make conda-lock.yml on a schedule and figuring out if packages changed.

One thing with allowing dependabot to keep doing the updates is it'll make one PR for every individual dependency that changes, which will be good for isolating changes, but it'll be a lot of PRs, and every one of them will need the lockfile to be regenerated. I was thinking that we'd maybe run update-lockfile on a weekly schedule, in which case it would update all of the versions together once a week. And then we'd also need to either run it manually or trigger it whenever pyproject.toml is changed to catch newly added or removed or changed dependencies.

With our ~500 direct+indirect dependencies, I don't think there will ever be a week in which none of them release a new version.

In removing the upper bounds I was thinking just of removing the ones where there's no known problem and the upper bound is just a safety measure that keeps the dependency from changing too much without some intervention & testing on our part. For pydantic>=2.0 and other known incompatibilities we'd leave them in so conda-lock only chooses from the versions that we expect could work.

One thing I think probably want to try and avoid is merge conflicts in conda-lock.yml... it's meant to reflect a single run of conda-lock and it's tens of thousands of lines long and not particularly human-friendly. Not sure what this means for the workflow we want to choose though.

I wonder if it’s possible to create an action that runs make conda-lock.yml any time the dependencies in pyproject.toml changes and commits the new files directly to the PR that changed pyproject.toml. This way the lock files are always updated when dependabot or a contributor bumps a version in pyproject.toml.

I think this should definitely be doable, in the same way that pre-commit.ci currently fixes any issues that make it to GitHub.

What are the benefits of installing from catalystcoop.pudl package instead of from the the repository directly? Distributing the package in PyPI and condo-forge seems like it will create unnecessary overhead.

I think we've really gone to an "application" model at this point, where catalystcoop.pudl isn't meant to be installed outside of its locked environment, and is intended for deployment to run the ETL so probably distributing these packages and doing the package isolation in testing is no longer particularly useful. I think for reproducibility of a particular run we'll use tags in git, or if something catastrophic happens, there are zipfiles archived on Zenodo automatically every time we tag. We should note this somehow so folks aren't confused by how stale the package is on PyPI and conda-forge.

Great point! It doesn't make much sense anymore to have optional dependencies if we treat PUDL like an application with pinned dependencies that produces data outputs. Does it make sense to keep the optional dependencies around until our main users no longer depend on install PUDL as a package?

I think maybe we want to consolidate all the currently "optional" dependencies and make them required, but what do we want to do about dependencies that are for interactive local development? Like the Docker container doesn't need to have JupyterLab installed in it, but we all probably want it. Should we just leave that up to individuals? Or include them in the lockfile? Or I guess, don't include them in the lockfile but include them in the rendered environment files that we'll use to install the environment locally with the new make target?

Oh, the downside of using conda to install google-cloud-sdk is that it's not available for win-64 or linux-aarch64. I don't know if this will end up mattering, but I got a warning at some point about not being able to build the docker container on my Mac because there was another package that wasn't available for aarch64...

With our ~500 direct+indirect dependencies, I don't think there will ever be a week in which none of them release a new version.

I thought dependabot only updates direct dependencies? If so, our dependabot PR frequency wouldn't change right? If it opens a PR for every indirect dependency update I think we should all the dependencies on a schedule.

I think we've really gone to an "application" model at this point, where catalystcoop.pudl isn't meant to be installed outside of its locked environment, and is intended for deployment to run the ETL so probably distributing these packages and doing the package isolation in testing is no longer particularly useful. I think for reproducibility of a particular run we'll use tags in git, or if something catastrophic happens, there are zipfiles archived on Zenodo automatically every time we tag. We should note this somehow so folks aren't confused by how stale the package is on PyPI and conda-forge.

I'm in favor of no more package releases. Can't think of a reason why someone would want to install PUDL given we're only distributing data now.

I think maybe we want to consolidate all the currently "optional" dependencies and make them required, but what do we want to do about dependencies that are for interactive local development? Like the Docker container doesn't need to have JupyterLab installed in it, but we all probably want it. Should we just leave that up to individuals? Or include them in the lockfile? Or I guess, don't include them in the lockfile but include them in the rendered environment files that we'll use to install the environment locally with the new make target?

This sounds like a good plan to me. So have two targets: make local_env which includes interactive development packages and make ci_env which has the minimum required packages?

I thought dependabot only updates direct dependencies? If so, our dependabot PR frequency wouldn't change right? If it opens a PR for every indirect dependency update I think we should all the dependencies on a schedule.

The dependabot only looks at our direct dependencies (in pyproject.toml but the lockfile resolves all of our indirect dependencies too. I was just saying that if we recreate the lockfile once a week, I think it will always have changes, probably like... 20 of them every week, and all in a single PR. While with dependabot we'll get several PRs -- one for every single direct dependency that changed, and since they'll all change pyproject.toml, we'll need to regenerate the lockfile every time dependabot makes a change, but there's really no point to that -- except that it might be easier to isolate which dependency is responsible for breakage when it happens (if it's a direct dependency anyway). But I think our problem has more often been that indirect dependencies change, and we have no idea what it was that changed or which of our dependencies relied on the thing that changed.

Would an update-conda-lockfile action that regenerates the lockfile and re-renders the environment files any time pyproject.toml is changed (including by dependabot) and auto-merges the new lockfile and environment files into whatever PR includes the changes to pyproject.toml end up having a lot of unpleasant conflicts in the big lockfile? We don't ever really want to merge the lockfile or environment files. We want them to be a unified reflection of a solved environment that flows from a particular pyproject.toml.

Updating all of the dependencies weekly sounds simpler than opening multiple dependabot PRs 👍

When would the lock and environment files have merge conflicts? Contributor updates pyproject.toml on a feature branch, update-conda-lockfile merges new conda lock and environment files into dev, contributor merges dev into feature branch and has to resolve merge conflicts in the lock and env files?

Hmm, @jdangerx it looks like we got another stochastic failure on the hypothesis test:

FAILED test/unit/io_managers_test.py::test_filter_for_freshest_data - hypothesis.errors.Flaky: Hypothesis test_filter_for_freshest_data(df=
      entity_id                          date utility_type              publication_time  int_factoid  float_factoid str_factoid
    0         5 1969-12-31 23:59:59.999999999        other 1969-12-31 23:59:59.999999869          -65  -1.415102e+15            
    1         7 1970-01-01 00:00:00.000000025          gas 1969-12-31 23:59:59.999999743           -6   5.130000e+02            
    2         M 1972-04-14 18:37:52.853935361        total 1969-12-31 23:59:59.999999994         -108   2.814750e+14            
) produces unreliable results: Falsified on the first call but did not on a subsequent one
Falsifying example: test_filter_for_freshest_data(
    df=
          entity_id                          date utility_type              publication_time  int_factoid  float_factoid str_factoid
        0         5 1969-12-31 23:59:59.999999999        other 1969-12-31 23:59:59.999999869          -65  -1.415102e+15            
        1         7 1970-01-01 00:00:00.000000025          gas 1969-12-31 23:59:59.999999743           -6   5.130000e+02            
        2         M 1972-04-14 18:37:52.853935361        total 1969-12-31 23:59:59.999999994         -108   2.814750e+14            
    ,
)
Found these contexts (['entity_id', 'date', 'utility_type']) in input data:
  entity_id                          date utility_type
0         5 1969-12-31 23:59:59.999999999        other
1         7 1970-01-01 00:00:00.000000025          gas
2         M 1972-04-14 18:37:52.853935361        total
The freshest data:
  entity_id                          date utility_type              publication_time  int_factoid  float_factoid str_factoid
1         7 1970-01-01 00:00:00.000000025          gas 1969-12-31 23:59:59.999999743           -6   5.130000e+02            
0         5 1969-12-31 23:59:59.999999999        other 1969-12-31 23:59:59.999999869          -65  -1.415102e+15            
2         M 1972-04-14 18:37:52.853935361        total 1969-12-31 23:59:59.999999994         -108   2.814750e+14            
Paired by context:
                                                               publication_time_in  int_factoid_in  float_factoid_in str_factoid_in          publication_time_out  int_factoid_out  float_factoid_out str_factoid_out _merge
entity_id date                          utility_type                                                                                                                                                                        
5         1969-12-31 23:59:59.999999999 other        1969-12-31 23:59:59.999999869             -65     -1.415102e+15                1969-12-31 23:59:59.999999869              -65      -1.415102e+15                   both
7         1970-01-01 00:00:00.000000025 gas          1969-12-31 23:59:59.999999743              -6      5.130000e+02                1969-12-31 23:59:59.999999743               -6       5.130000e+02                   both
M         1972-04-14 18:37:52.853935361 total        1969-12-31 23:59:59.999999994            -108      2.814750e+14                1969-12-31 23:59:59.999999994             -108       2.814750e+14                   both
Unreliable test timings! On an initial run, this test took 305.40ms, which exceeded the deadline of 200.00ms, but on a subsequent run it took 53.25 ms, which did not. If you expect this sort of variability in your test timings, consider turning deadlines off for this test by setting deadline=None.

You can reproduce this example by temporarily adding
@reproduce_failure('6.88.1', b'AXicNYzbDYAwDMTslLYbIDEMH+zDFuzFJAxDWoSly10eClJBUwUXuLDKEWm19+B+kM682UwYxSKNn8gh60iyZ1ON+atpl/NbfLz5gAOv')
as a decorator on your test case

It also seems like a lot of the dates that show up in these tests are suspiciously close to (within a few nanoseconds of) Unix time zero.

When would the lock and environment files have merge conflicts? Contributor updates pyproject.toml on a feature branch, update-conda-lockfile merges new conda lock and environment files into dev, contributor merges dev into feature branch and has to resolve merge conflicts in the lock and env files?

I think any time there are changes to conda-lock.yml from more than one branch that run into each other we want to regenerate the locked outputs from scratch, based on whatever is found in pyproject.toml which really does need to deal with merge conflicts. This would potentially happen frequently if we were using dependabot, since it typically makes several PRs simultaneously, updating different dependencies individually, which then all get merged into dev in quick succession.

In the conda-lock repo itself they just re-lock the dependencies on a schedule.

I think we're trying to avoid a scenario where, say, we run a nightly build in an environment that's determined by a lockfile that doesn't actually correspond to the the pyproject.toml file associated with the commit. Though I guess if we're good about only ever building the environment from the lockfile, and installing pudl using pip --no-deps this isn't a catastrophe -- the lockfile that was checked into git would define the environment used for the build, and it would be there in git for future reproducibility. Even if the lockfile it self wasn't reproducible because it was some weird frankenstein merge.

• Started make nuke again locally at 10:10am
• Finished at 12:13pm
• need to rm pudl.sqlite && alembic upgrade head when running nuke.
• Started make nuke at 13:35
• Finished at: 15:50

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (6596401) 88.7% compared to head (071766a) 88.7%.
Report is 5 commits behind head on dev.

@@          Coverage Diff          @@
##             dev   #2968   +/-   ##
=====================================
  Coverage   88.7%   88.7%           
=====================================
  Files         90      90           
  Lines      10994   10994           
=====================================
  Hits        9758    9758           
  Misses      1236    1236           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pip not picking up on the version of recordlinkage isn't breaking anything... but it means I can't specify a version for recordlinkage in pyproject.toml or it will try to install it, even though it's already satisfied by the conda environment. I guess this wouldn't happen any more since I'm doing pip install --no-deps now, but it seemed like an annoying mixed-environment gotcha.